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OVERVIEW 


ostsecondary education is widely seen as a necessity in the modern economy, yet among low- 
and middle-income families, college enrollment rates are dismayingly low — and graduation 
rates are even lower. College Promise programs, which cover local students’ college tuition and 
fees, are one strategy states and municipalities use to help. College Promise programs aim to 
put college in students’ reach financially and to institute a college-going culture for all K-12 students. 
But traditionally, these programs look only to expand college access, not to address college success. 


Detroit’s Promise program was designed to encourage college attendance among some of the na- 
tion’s most underserved students, those in Detroit, Michigan. The next step was to help students 
succeed once they enrolled in college. To do so, MDRC and the Detroit Promise partnered to create 
the Detroit Promise Path, an evidence-based student services program. Detroit Promise Path stu- 
dents begin meeting with college coaches in the late summer before their first semester of college. 
They are given an incentive to attend coaching meetings in the form of a monthly gift card refilled 
with $50 each month that they meet with coaches as directed. The program lasts all year, including 
summer semesters, when students are encouraged to enroll in summer classes or engage in a local 
summer jobs program. The entire operation is supported by a management information system. 


This report presents findings from MDRC’s randomized controlled trial evaluation of the Detroit 
Promise Path. About two-thirds of eligible students were randomly assigned to be offered the new 
program, while the rest were assigned to a control group who receives the Promise scholarship alone, 
and thus does not meet with coaches or receive incentives. Comparing the two groups’ outcomes 
over time provides a reliable estimate of the effects of the Detroit Promise Path. The findings in this 
report include the following: 


¢ The program has a positive effect on students’ persistence in school, full-time enrollment, and 
credit accumulation. The effects on persistence and full-time enrollment in the second semester are 
statistically significant and among the largest MDRC has found in postsecondary experiments. They are 
especially notable because less than two-thirds of the study sample enrolled during the first semester. 


¢ Although it is too early to reach a conclusion about effects in the second year of the study, the 
early findings are encouraging. Data on second-year outcomes are only available for the first 
group of enrollees, who have been followed for two years. Findings in the second year are positive, 
but not all are statistically significant and the effect on credits earned is smaller. 


¢ Participation rates were high among enrolled students, and students reported positive experi- 
ences in the program, especially in their relationships with their coaches. The vast majority of 
program group students engaged with the program, and the average number of coaching meetings 
remained high throughout the two program years. In focus groups and a student survey, nearly 
all students reported valuing the program. 


A future report will examine the program’s effect on graduation rates. It is clear, though, that Detroit 
Promise Path is having a positive effect on students in the first two years. This evaluation shows 
that building student support services into Promise scholarships can have a meaningful effect on 
students’ academic progress. 
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ostsecondary education is widely seen as a necessity in the modern economy, yet in low- and 
middle-income families, college enrollment rates are dismayingly low — and graduation 
rates are even lower.’ College Promise programs are one strategy states and municipalities 
use to help students pay for college. These programs, which have received bipartisan support 
from policymakers and which now number more than 300 nationwide, provide financial support 
to cover local students’ college tuition and fees. College Promise programs aim to put college 
in students’ reach financially and to institute a college-going culture for all K-12 students. But 
traditionally, programs that aim to expand college access have not also advanced college success. 


Detroit’s Promise program was designed to encourage college attendance among some of the na- 
tion’s most underserved students, those in Detroit, Michigan. The next step was to help students 
succeed once they enrolled in college. To do so, MDRC and the Detroit Promise partnered to 
create the Detroit Promise Path, an evidence-based student services program. Poverty rates are 
high and educational attainment rates are low in the Detroit area — fewer than 15 percent of city 
residents possess a bachelor’s degree or higher.” Local students face a variety of systemic barriers 
to success when attempting higher education. The Detroit Promise Path program aims to address 
these barriers by helping students manage financial and nonfinancial obstacles alike. 


This report presents interim findings from MDRC’s experimental evaluation of the Detroit 
Promise Path showing that the program improves students’ enrollment rates and overall credit 
accumulation. Over the two-year follow-up period, the program has a positive impact on semester- 
to-semester persistence in college, full-time enrollment, and credit accumulation. However, for 
the subset of students in their second year of the study, the impacts become more modest over 
time. It is not yet clear whether the program will have a measurable impact on graduation rates, 
but the program is having a positive impact on students in the first two years. 


DETROIT PROMISE PATH: ADDRESSING COLLEGE 
ACCESS AND SUCCESS 


The Detroit Promise, administered by the Detroit Regional Chamber, was launched in 2013 as 
the Detroit Scholarship Fund to help more of the city’s high school graduates enroll in college. 
It covers any difference between a student’s financial aid and tuition for up to three years of at- 
tendance. To be eligible, the student must have graduated from a Detroit high school and be a 
resident of the city of Detroit. (In 2017, the program expanded to help students with qualifying 
ACT or SAT scores and qualifying high school grade point averages pay tuition at public four-year 
colleges in Michigan. These students are not part of the present study.) A student can enroll in 
the scholarship within one year of finishing high school and is eligible for funds for up to three 
years. Students are directed to enroll in school full time, though this requirement is not enforced, 
meaning that students do not lose the scholarship if they drop below full-time status. 


1 National Center for Education Statistics (2019). 
2 U.S. Census Bureau (2018). 
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Chamber staff members observed that the scholarship was helping more high school graduates 
enroll in college initially; however, large numbers of Detroit Promise recipients were dropping 
out of college before their second year. Program staff members wanted to incorporate student 
success components into the scholarship so that Promise students would not only enroll in school 
but would be likely to succeed there. To do so, the Chamber and MDRC partnered to create the 
Detroit Promise Path, which adds four components to the existing scholarship program (see 
Figure 1). 


FIGURE 1. SCHOLARSHIP OTHER SERVICES AND BENEFITS 
Detroit 
Promise =| 
Path = 
Program Covers Offers Offers $50 Engages Uses a 
Model tuition campus each month students in management 
and fees coaching for expenses the summer information 
after qs and requires not covered by by encouraging system 
financial aid each student financial aid, enrollment or to track student 
for students to meet with such as bus connecting participation 
enrolled full time a coach twice passes or them them to and automate 
and assists per month books, to local initiatives appointments 
them with students who such as and financial 
summer meet with summer job incentives 
tuition coaches programs for better 
monitoring 


The core of the program is its coaching component. Detroit Promise Path students begin meeting 
with coaches in the late summer before their first semester of college. They are given an incentive 
to attend coaching meetings in the form of a monthly gift card refilled with $50 each month they 
meet with coaches as directed. This financial incentive helps students pay expenses not covered 
by financial aid. The program lasts all year, including summer semesters, when students are en- 
couraged to enroll in summer classes (paid for by the scholarship) or engage in a local summer 
jobs program called Grow Detroit’s Young Talent. The entire program operation is supported 
by a management information system — Microsoft Dynamics 365 — that coaches use to track 
participation in coaching sessions, email and phone outreach, and text messages. (More detail 
on the program model and the implementation of the program can be found in a 2018 MDRC 
publication, “Learning from Success: The Detroit Promise Path.”)® 


The Design and Evaluation of Detroit Promise Path 


MDRC and the Chamber partnered to design this new program based on strategies drawn from 
rigorous research conducted in community colleges by MDRC and others in the field. Multiple 


3  Ratledge and Vasquez (2018). 
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experimental studies have shown that approaches such as enhanced advising and financial 
incentives can have positive, but modest effects.* However, evaluations of programs that com- 
bine multiple, evidence-based interventions and provide services for a longer time have shown 
larger effects.* Most notably, the City University of New York designed and implemented its 
Accelerated Study in Associate’s Programs, which was the first to show through a random as- 
signment evaluation that comprehensive student support programs could dramatically increase 
graduation rates.® 


The Detroit Promise Path was specifically designed to meet the needs of Detroit’s students, based 
on the Chamber’s knowledge of the local student population. MDRC provided extensive technical 
assistance during the first two years to set up the program, develop cost-effective management 
strategies, train staff members how to use the management information system to track student 
participation, and ensure the program was operating as intended. The current study includes 
students at five Detroit-area community colleges: Henry Ford College, Macomb Community 
College, Oakland Community College, Schoolcraft College, and Wayne County Community 
College District. The program is operated centrally by the Chamber with coaches housed at each 
campus to meet with students in person during school hours. 


MDRC’s evaluation of the Detroit Promise Path uses a randomized controlled trial design, widely 
considered the gold standard in social science research. This experimental design makes it pos- 
sible to estimate the impact of the new program’s components on students’ academic outcomes. 
Eligible students were randomly assigned to either a program group, in which campus coaches 
made contact with students and students were eligible for the additional financial incentives, or 
a control group, in which students continued to receive their Promise scholarships and college 
services but did not receive outreach from coaches or the monthly financial incentives. Random 
assignment is a fair way to distribute the limited number of spots in the program, and it also allows 
unbiased estimation of the program’s impacts. Because students were assigned randomly to the 
two groups, program and control group students should be similar at the start of the study with 
respect to both observable characteristics such as sex and race and unobservable characteristics 
such as motivation and tenacity. As a result, any statistically significant difference in outcomes 
measured between the two groups later can be reliably attributed to the program. 


To be clear, all students continue to be eligible for the Promise scholarship; it is the additional, new 
program components that are being tested. Control group students continue to receive scholar- 


4 Welbeck, Ware, Cerna, and Valenzuela (2014); Castleman and Page (2016); Mayer, Patel, Rudd, and Ratledge 
(2015); Sommo, Mayer, Rudd, and Cullinan (2012); Bettinger and Baker (2014); Scrivener and Coghlan (2011); 
Barnett et al. (2012); Scrivener and Weiss (2009). 


5 Barr and Castleman (2016); Evans, Kearney, Perry, and Sullivan (2017); Carrell and Sacerdote (2017); Rolston, 
Copson, and Gardiner (2017). 


6 Boykin and Prince (2015); Scrivener et al. (2015). 
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ship funds as they would have in the absence of the study.’ Although the Detroit Promise does 
reach out to students while they are still in high school to inform them about the scholarship, 
the Detroit Promise Path program evaluated in this report only interacts with college students 
after they enroll. 


Student Population 


Table 1 presents the demographic characteristics of the participants in the study.® Participants’ 
average age is 18, as the program serves recent high school graduates.® The vast majority identify 
as people of color and most do not live with a parent who has completed a college degree, reflect- 
ing the population of the city of Detroit." 


The federal government reports that nearly half of residents under the age of 18 in Detroit live in 
poverty, while the United Way’s Asset Limited, Income Constrained, Employed (ALICE) reports 
that over 70 percent of Detroit’s children do." At the five participating colleges, rates of federal 
Pell Grant receipt for first-time, full-time students range from 27 percent to 81 percent, with the 
highest percentages at the two colleges enrolling the greatest numbers of Detroit Promise recipi- 
ents. In short, this program serves a student population experiencing a high risk of not attending 
or completing college. 


7 Most Detroit Promise students do not need substantial scholarship coverage for tuition and fees. Most 
qualify for federal, need-based Pell Grants that fully cover the cost of tuition and fees in community college. 
Proponents of Promise programs argue that simply offering the scholarship may make students more likely 
to enroll since many low-income students do not realize that they are eligible for need-based financial aid. 
The idea is that even for students who do not receive any scholarship dollars, the “promise” of free tuition 
may relieve their financial anxiety and make them believe college is financially in reach. 


8 Because the study was embedded into the typical application process for Detroit Promise, very few baseline 
demographic characteristics were collected from students and are available for analysis. 


9 The Chamber also collects baseline data on students’ ACT and SAT scores. Test score is included as a 
covariate in the impact analysis (as prespecified in an analysis plan); however, it is not presented in this table 
because the test data collected changed from ACT to SAT between the first and second cohorts of students 
to enter the study (that is, the first and second waves or classes of students to enter), and because score 
data are missing for many students (over 50 percent). Among those for whom scores are available, there are 
no statistically significant differences between the program and control groups (p = 0.42 and p = 0.17 for 
ACT and SAT scores, respectively). 


10 Most students in the study are the first in their families to attend college. The Detroit Promise application 
asks students whether they live with a parent who has completed a four-year college degree, which is 
slightly different from the general definition of “first in the family to attend college.” In the student survey 
(reported below), however, students were asked whether they had any family members who had completed 
college (a broader question than the application’s). Among the first group of students to enter the study, 
only about one-third of students responding to the survey reported having any family member who had 
completed a degree, while one-quarter had relatives who had attended college and not finished, and 20 
percent had no family members who had been to college at all. 


11 United Ways of Michigan (2017); U.S. Census Bureau (2018). 
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TABLE 1. Selected Baseline Characteristics of 
ed aeyele-lanme-lalem Oxelalige) Mel aelelom i (-Van)el-1 a) 


PROGRAM CONTROL 


CHARACTERISTIC GROUP GROUP DIFFERENCE P-VALUE 
Female (%) 58.3 60.6 -2.3 0.432 
Male (%) 41.7 39.4 2.3 0.432 
Age 17.9 17.9 0.0 0.944 
Race (%) 

Black or African-American 80.3 80.6 -0.4 0.880 
Hispanic or Latino? 12.6 11.0 1.5 0.432 
Two or more races 4.3 3.8 0.5 0.675 
Asian, Native Hawaiian, or other Pacific Islander 1.7 2.1 -0.5 0.539 
White 1.2 1.6 -0.4 0.546 
American Indian or Alaska Native 0.0 0.8 -0.8 0.012 
Does not live with a parent who has earned 

a bachelor's degree (%) 81.1 77.8 3.3 0.165 
Sample size (total = 1,268) 829 439 


SOURCES: MDRC calculations from Detroit Regional Chamber Promise application data. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 


KK 


Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Items are shown as “missing” only if more than 5 percent of the group was missing data. 

Weights are calculated to make the effective (weighted) random assignment ratio the same in all random 
assignment blocks. The effective random assignment ratio is equal to the full sample’s random assignment 
ratio. 

SAT and ACT score are not included because more than half of all test score data are missing. SAT score 
was added to the Detroit Regional Chamber Promise application in the second year of the study, at which time 
both test score questions became optional. Among students with scores recorded, there are no statistically 
significant differences between the program and control groups (p = 0.42 for SAT, p = 0.17 for ACT). 

*Hispanic or Latino students may be of any race. 


Study Enrollment 


Study enrollment took place in summer 2016 and summer 2017, ahead of the fall 2016 and 2017 
semesters. All students who were eligible for the Detroit Promise scholarship were eligible for the 
new Detroit Promise Path program. This report presents first-year findings for all study students. 
The second-year findings are shown only for the first cohort: the group who enrolled ahead of fall 
2016, and who therefore have the longest follow-up period. MDRC’s final report on the project 
will present three-year findings for the full sample. 


This study uses an intent-to-treat analysis. That is, it compares the outcomes of all students who 
were offered Detroit Promise Path (the program group), whether or not they participated in Detroit 
Promise Path, with the outcomes of all students who were not offered Detroit Promise Path (the 
control group). This context is important for interpreting the results, because around one-quarter 
of the students who were offered Detroit Promise Path never enrolled in college, received very 
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limited Detroit Promise Path services, and probably did not benefit from the program, but are 
still included in analyses. These nonenrollees received initial email outreach from the Detroit 
Promise Path coaches as well as some follow-up text messages; however, they did not receive the 
bulk of Detroit Promise Path program services. Because such a large proportion of students in 
the program group did not enroll in college or meaningfully interact with the program, it is also 
of interest to understand the impact of Detroit Promise Path on those program group students 
who did enroll in college and received more than just the initial outreach from Detroit Promise 
Path coaches. This supplementary analysis is presented in Appendix A. 


In any intent-to-treat analysis, a certain number of program group members may not receive 
services. The proportion is relatively high in this study for two reasons. First, in most of MDRC’s 
studies in postsecondary education, eligible students are actively recruited before agreeing to 
participate in the program and study. But this was an opt-out study, meaning that all Detroit 
Promise applicants were randomly assigned unless they asked not to be during the application 
and informed-consent process. Consequently, in this study, participants did not have to express 
interest in the add-on services of Detroit Promise Path to join. A major benefit of this approach 
is that the study results are more generalizable, meaning that they better represent what other 
programs or colleges implementing such a program might expect to happen. The approach also 
met two of the Chamber’s goals: to make sure that all students were included, rather than just 
those who sought out assistance, and to make the program sign-up process as easy as possible for 
students. A potential drawback of the opt-out approach is that because students did not actively 
elect to be part of the study and the new program, some students were not interested and therefore 
were less likely to engage in the program. 


The second reason a relatively high proportion of program group members did not interact with 
the program relates to the timing of study enrollment. In most of MDRC’s studies in postsecondary 
education, eligible students have already matriculated when they are randomly assigned — that 
is, they have been admitted to college and have often even selected classes. In contrast, because 
random assignment in this study began in the early summer and continued through September, 
many students signed up for the program before making final decisions about college enrollment. 
Students were asked on the Promise application where they would be attending college, but many 
of them had not finished the matriculation process when they completed that application. 


Because of these two elements of the study design, the sample includes a large number of students 
who signed up for the scholarship but never actually enrolled in college. These students’ outcomes 
are all reported in the analyses as zeroes, as they are not enrolled or accumulating credits. These 
zeroes draw the overall outcomes downward. (Though if the students enroll in the future, they 
will be reflected in the data.) As mentioned aboved, Appendix A shows the analysis without these 
students who never enrolled. 


IMPACTS ON ACADEMIC OUTCOMES 


In the first year of the program, Detroit Promise Path has positive impacts on program group 
students’ enrollment and full-time enrollment rates, and on the proportion of students who 
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completed 24 or more credits — in other words, a full-time course load — in their first year. 
These impacts are large and statistically significant. Turning to second-year outcomes, which are 
currently only available for the first cohort, enrollment impacts become more modest over time. 
More detail on each outcome is presented below and in Appendix Table A.1. 


The analyses of the program’s impacts on academic outcomes draw on two data sources: the 
National Student Clearinghouse (NSC) and the five Detroit-area community colleges in the study. 
The analysis uses NSC data for enrollment outcomes, because NSC data capture enrollment at 
almost any college nationwide, not just those in the study. Students who enrolled at colleges outside 
of the study are therefore still reflected in the data on overall enrollment. For outcomes related to 
credits and full-time enrollment, however, only students at the five study colleges are included. 


Increased Enrollment and Persistence in School 


Figure 2 displays enrollment and full-time enrollment during the first year after students were 
randomly assigned. In the first semester, the program’s estimated impact on enrollment is 5 
percentage points.’* Its estimated impact on enrollment grows to about 8 percentage points in 
the second semester. 


Detroit Promise Path’s estimated impact on full-time enrollment for the full study sample increases 
from about 6 percentage points in the first semester to about 10 percentage points in the second 
semester.'* This finding shows that there is a sizable group of students who currently enroll part 
time but would enroll full time with direction and support. 


Detroit Promise Path coaches strongly encouraged program group students to participate in 
some productive activity over the summer, such as enrolling in summer classes, working in jobs 
related to their majors, or joining the Grow Detroit’s Young Talent youth employment program." 
Summer course enrollment is an activity that increases students’ progress toward a degree and 
that has been correlated with college completion. There was an estimated 14 percentage point 
increase in summer course enrollment during the first summer after random assignment (an 
increase from 7 percent to almost 21 percent — see Table 2) among those who joined the study 
in 2016. This impact is larger than the impact on enrollment in fall or spring, and it is among 
the largest impacts on summer enrollment recorded in a postsecondary experiment — part ofa 


12 This estimate is significant at the 0.10 level, which means that if the true impact on enrollment were zero, 
there is less than a 10 percent chance of observing an estimated impact this large. 


13 MDRC published early findings in 2017 that present impact estimates for the first study cohort only. See 
Ratledge (2017). The present report includes all study students, so numbers are not consistent across these 
two publications. In addition, there are some data updates to the first study cohort that slightly change 
those estimates. These data changes reflect updates to colleges’ administrative data sets made later in the 
spring 2017 semester, and include instances where students switched levels in the developmental (remedial) 
math sequence mid-semester, withdrew for medically excused reasons, or withdrew from late-starting 
courses. 


14 Grow Detroit’s Young Talent is a six-week summer jobs program for Detroit residents between the ages of 
14 and 24. Coaches connected interested students to the program if they were not going to take summer 
courses. 
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FIGURE 2. Detroit Promise Path Increases Enrollment 
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SOURCES: MDRC calculations using data from the National Student Clearinghouse 
and the Detroit Promise Path colleges. 


NOTES: A two-tailed t-test was applied to differences between research groups. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; 
* = 10 percent. 

Estimates are adjusted by site, cohort, gender, ACT or SAT score, and the interaction 
of gender and race/ethnicity. 

Weights are calculated to make the effective (weighted) random assignment ratio the 
same in all random assignment blocks. The effective random assignment ratio is equal 
to the full sample’s random assignment ratio. 

Full-time enrollment is defined as enrollment in 12 or more credits at a student’s 
college of random assignment. 


growing body of evidence that encouraging summer enrollment can be an important component 
of a wide variety of interventions.’® Detroit Promise summer funding is also available to control 
group students, but the encouragement and support from coaches seems to lead more students 
to take advantage of that funding. 


15 Attewell (2013); Headlam, Anzelone, and Weiss (2018); Liu (2016); Patel and Rudd (2012); Scrivener et al. 
(2015); Sommo and Ratledge (2016). 
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TABLE 2. Enrollment and Credit Outcomes During the Summer 


PROGRAM CONTROL DIFFERENCE STANDARD 
OUTCOME GROUP GROUP (IMPACT) ERROR P-VALUE 
First summer semester 
Enrolled (%) 20.5 7.0 13:5: 24% 2.6 0.0000 
Credits attempted 1.0 0.4 0.7 *** 0.2 0.0000 
Credits earned 0.8 0.3 0.5 *** 0.2 0.0020 
Sample size (total = 624) 415 209 


SOURCES: MDRC calculations using data from the NSC and the Detroit Promise Path colleges. 


NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site, cohort, gender, ACT or SAT score, and the interaction of gender and race/ 
ethnicity. 

Weights are calculated to make the effective (weighted) random assignment ratio the same in all random 
assignment blocks. The effective random assignment ratio is equal to the full sample’s random assignment 
ratio. 


NSC data are used for overall enrollment. Credits attempted and credits earned are based on data from a 
student's college of random assignment. 


Detroit Promise Path’s first cohort completed their second year of the program in spring 2018. 
For this half of the study sample (589 students), the program’s impact on enrollment in their 
second year is roughly similar to the impact for the full sample’s first year (see Appendix Table 
A.1). In the third program semester (fall 2017), program students were nearly 8 percentage points 
more likely to enroll than control group students. In the fourth program semester, the estimated 
impact of the program is no longer statistically significant but, at 5 percentage points, is still of 
a meaningful size. 


For this same first cohort, the program’s impact on full-time enrollment seems to decrease in 
the second year. In the third program semester, the estimated impact of Detroit Promise Path 
on full-time enrollment is about 5 percentage points, and is no longer statistically significant 
at conventional levels. The estimated impact of the program is similar in the fourth program 
semester, and the impact regains significance at the 0.10 level. 


The NSC data reveal that a substantial share of Detroit Promise Path students — roughly 1 in 10 
in any given semester — enrolled at colleges other than the ones where they said they planned to 
attend when they entered the study (see Appendix Table A.1). This proportion increases slightly 
over time. These students are missing from the data for the measures of full-time enrollment and 
credit accumulation. As a result, the average outcomes for full-time enrollment are underestimated 
(as are the measures of credits earned, discussed below). That said, these underestimates affected 
the program and control groups equally: Program and control students enrolled at similar rates at 
other colleges than the ones they had selected at the time of random assignment. The estimated 
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impact on full-time enrollment is therefore probably unaffected. In other words, the overall 
full-time enrollment averages for the program and control groups are higher in reality, but it is 
unlikely that the difference between the two research groups is underestimated. 


Increased Credits Earned 


In their first year, program group students earned an average of 1.7 credits (both developmental 
and college-level) more than control group students, a 25 percent increase that is statistically sig- 
nificant. (For context, most courses at these colleges are worth three or four credits.) By the end 
of the second year (for which data are available only for the first cohort), program group students 
had earned an average of 2.4 more credits than their control group counterparts (see Appendix 
Table A.1). This impact is also statistically significant. However, all study participants appear to 
have earned fewer credits on average in the second year, as higher percentages of them enrolled 
at nonstudy schools (from which data are not available) or did not enroll at all. 


It is also informative to look at credit accumulation categorically, which makes it possible to 
compare the proportion of students in the program and control group who successfully com- 
pleted full-time, part-time, or less than part-time course loads, or who earned no credits. Figure 3 
presents cumulative total credits earned in the first year of the program (see also Appendix Table 
A.3). There is a 9 percentage point impact on earning any credits. In other words, the program 
leads more students to have earned at least some credits. More than half of this impact comes 
from students earning 24 or more credits, which is equivalent to a full-time course load. Almost 
11 percent of program group students accumulated 24 credits or more, compared with about 
6 percent of control group students. The 5 percentage point impact on the accumulation of 24 
credits or more means that program group students are about twice as likely to have completed 
full-time course loads in their first year. 


As noted with full-time enrollment, overall credit accumulation averages are underestimated 
here because high percentages of students enrolled in colleges other than the ones they said they 
planned to attend at the time of random assignment, yet data on total credits earned are only 
available for the schools they did name at that time. It is unlikely that the impact of the program 
on credit accumulation is affected by these missing data. 


The final report on the project will present the program’s estimated impact on degrees earned 
for both cohorts over three years. The data on degrees earned will confirm whether the Detroit 
Promise Path has a long-term positive impact on graduation rates. 


PROGRAM IMPLEMENTATION 


The research team conducted two rounds of qualitative research in spring 2017 and spring 2018 
to understand how the program was being implemented: how faithful the program as imple- 
mented was to its intended model, how students were experiencing the program, and how much 
contrast existed between the college experiences of program group and control group students. 
This research effort included focus groups and interviews with study sample members (including 
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FIGURE 3. Credit Accumulation in the First Year of the Study 
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SOURCE: MDRC calculations using data from the Detroit Promise Path colleges. 


NOTES: A two-tailed t-test was applied to differences between research groups. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; 
* = 10 percent. 


Estimates are adjusted by site, cohort, gender, ACT or SAT score, and the 
interaction of gender and race/ethnicity. 


Weights are calculated to make the effective (weighted) random assignment ratio 
the same in all random assignment blocks. The effective random assignment ratio is 
equal to the full sample’s random assignment ratio. 


both students who were participating and not participating in program services, and both those 
who were enrolled and were not enrolled in school); interviews with staff members; a review 
of management information system data to measure program participation and the amount of 
program services students were receiving; and an online student survey fielded to both program 
and control group students. 


The program was well implemented by the Chamber’s staff. Interviews with program staff mem- 
bers and students indicated a high level of fidelity to the two primary components of the program 
model: coaching and incentives. Program participants understood the central requirement of the 
program (that they meet with a coach twice each month), and reported that coaches reached out 
to them in multiple ways, most often through phone calls and text messages. Coaches reported 
that they used the management information system extensively to track student participation, 
and the research team’s review of data from that system backed up their reports. The program 
manager overseeing the coaches produces reports summarizing data from the system regularly, 
and those reports were used by program staff members to review students’ progress and decide 
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which students needed more outreach. For example, staff members would send text reminders 
to all students seen fewer than twice in a given month, to remind them to make appointments. 


There was some variation in implementation among the colleges, in the implementation of coach- 
ing in particular. At one college where internal support for the program was far lower than the 
other schools, coaching sessions looked quite different: Fewer enrolled students participated 
in coaching sessions, and those who did participate met with coaches less often in person than 
students at other colleges, and more often by phone (or occasionally, by video). Colleges whose 
staff members and administrators were more supportive of the program also helped students and 
coaches more actively when it came to other issues — resolving problems related to financial aid, 
for example. In interviews, program group students enrolled at the college where internal support 
was not strong reported less interaction with the coach and with scholarship staff members, and 
more confusion about the college and program. This college was also the only one where pro- 
gram group members who had dropped out said that their coach had not tried to help them stay 
enrolled. A subgroup analysis of academic impacts by college (presented in Appendix Table A.2) 
shows that at this college (College 5 in the table), there are no discernable impacts on students’ 
academic outcomes."® 


Program Participation Rates 


Figure 4 presents the number of times program group members met with coaches. (Students re- 
ceived incentives if they attended coaching meetings, so attendance at coaching meetings encap- 
sulates overall participation.) Program group members were directed to meet with their coaches 
twice per month throughout the fall and spring semesters. In all four semesters, most enrolled 
students met with coaches six or more times each semester. On average, enrolled students met 
with their coaches about 5.3 times in the first semester, and between roughly 6.7 and 7.6 times 
in subsequent semesters. 


Likewise, the proportion of enrolled students who met with their coaches six times or more 
increased through the third semester and held steady in the fourth semester. While this increase 
is probably in part a result of the kind of students who stayed enrolled in college (for example, it 
may be that more motivated students stayed enrolled longer and were also more likely to abide 
by program directives to meet with coaches), these high participation rates show that program 
participants find the Detroit Promise Path program valuable, as they continue to stay engaged 
over the course of their time in college. (And in fact the survey findings reported below show 
that students did report they valued the program.) Among participants in the first cohort who 


16 The subgroup analysis of academic impacts by college shows Detroit Promise Path’s estimated impact at 
each college separately. The program has positive and statistically significant estimated impacts at three of 
the five colleges, and no statistically significant impact at the remaining two. However, there is no indication 
that the variation in impacts among the five colleges is statistically significant itself. In other words, the 
program may not have truly different impacts across the colleges; instead, the variation in estimated impacts 
across colleges may be due to chance. The small sample sizes at the college level limit the study’s ability to 
detect statistically significant impacts within colleges and statistically significant variation in impacts among 
colleges. 
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FIGURE 4. Coaching Session Attendance Among Enrolled Program Group Students 
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SOURCE: MDRC calculations using data from the Detroit Promise Path colleges and the MDRC 
management information system. 


NOTE: Coaching sessions for students who enrolled at colleges other than the ones they identified 
on their Detroit Promise scholarship applications are not included. 


remained enrolled through all four semesters, nearly three-quarters continued to meet with their 
coaches six or more times in the fourth semester. 


Student Experiences in the Program 


MDERC and the University of Michigan’s Youth Policy Lab fielded a survey to study participants in 
both the program and control groups, in order to learn more about their experiences with college 
and with Detroit Promise. The survey was fielded to sample members in their third semester after 
entering the study. Analysis for the second cohort is still in progress; the findings presented here 
are exclusively for the first cohort, who entered the study in fall 2016 and were surveyed in fall 
2017. The full study sample of students was surveyed, both students who were participating and 
not participating in program services, and both those who were enrolled and were not enrolled 
in school. Unfortunately, survey response rates were low: Only about one-third of each research 
group responded to the survey. As a result, the survey analysis will not report on statistically 
significant differences between experimental groups. Instead, the survey and focus group find- 
ings are presented together, highlighting themes and trends that complement the quantitative 
findings on academic outcomes. 
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One topic the survey covered was students’ use of support services. Students attending institu- 
tions with comprehensive support services are more likely to persist in and complete college, 
and are generally more satisfied with their college experiences."” The survey asked both control 
and program group members about their use of program and campus student services. Program 
group members said that they met frequently with their Detroit Promise Path campus coaches, 
as the model requires. In the focus groups, program group members reported that their interac- 
tions with coaches were very positive, widely agreeing that their coaches provided support and 
motivation. As one student said, “[The coach] keeps you on track and reminds you that you're 
trying to do something positive with your life.” 


Beyond just meeting with coaches, the early survey results indicate that program group members 
used campus resources outside the program more regularly than control group students did. For 
example, almost half of program group survey respondents reported meeting with academic 
advisers other than their coaches at least three times since starting college; only about a quarter 
of the control group survey respondents reported meeting with academic advisers that often. 
Control group respondents were also more likely to report never meeting with academic advisers 
than program group students. While approximately the same percentage of students in both 
groups reported seeing career advisers — slightly more than half in both groups — program 
group students reported going to see career advisers more often. These meetings are in addition 
to meetings with the coaches for the program group, indicating that the coaches are helping more 
students use existing campus resources and helping them do so more frequently. Coaches across 
all colleges said that they frequently — even daily — referred students to support services on 
campus, especially counseling and tutoring. On the survey, students confirmed that they made 
use of these services. 


The survey also asked students questions related to their level of engagement in the academic 
culture of the institution: about developing positive interactions with faculty, feeling comfort- 
able with college expectations, and forming effective study habits. Such engagement is especially 
important for underprepared students.’ Program group members were much more likely than 
control group members to report having a good understanding of college processes, including 
those related to financial aid, academic requirements, and selecting courses. Only about half of 
control group respondents indicated a high level of understanding of academic requirements, 
compared with about three-quarters of program group respondents. Program group respondents 
were also more likely to agree that they had the support necessary to succeed in college, including 
academic support and financial support. Given the high level of financial need in this student 
population, seeing a large difference in students’ evaluation of the financial support available to 
them is a highly encouraging finding. 


Program group members were also asked questions specific to the Detroit Promise Path program 


components that touched on their satisfaction with the coaching and financial incentives. The 
vast majority of program group members reported that participating in the program had been 


17 Kuh et al. (2008). 
18 Barbatis (2010). 
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valuable or very valuable. This finding aligns with the student participation rates described above, 
showing that the majority of enrolled students met with their coaches six or more times each 
semester. In the student focus groups, program participants reported that the coaching was the 
most valuable part of the program to them, though they also appreciated the monthly financial 
support. In the survey, program participants reported that they primarily used their gift cards for 
food, transportation, books, and supplies. In fact, a majority of respondents reported that they had 
used their gift cards to pay for food and nearly half reported using gift cards for transportation. 


In interviews, Detroit Promise Path coaches and program group students across the colleges re- 
ported that their main topics of conversation were financial aid, most commonly how to resolve 
financial aid issues like verification for the Free Application for Federal Student Aid (FAFSA);'° 
the development of soft skills such as time management; and external, nonacademic issues such 
as racism, family dynamics, transportation difficulties, and housing and food insecurity. One 
student described meetings with a coach as covering, “Everything. School, life, everything.” 
Several students said that they felt like their coaches understood them because they had also 
come from urban Detroit; they saw their coaches as trusted family members who could give them 
good advice. One male student explained that he connected with his male coach who was from 
the same neighborhood in the city as him, saying it meant a lot to see him be successful “because 
black males don’t get to go to school.” A few students said that their coaches helped build their 
confidence, for example by role-playing how to talk to faculty members during office hours, which 
they had never tried to do before, or by walking with them to the tutoring center when they were 
too nervous to go alone. 


While students said that they faced academic issues, they nearly universally said that nonacademic 
problems were even more of a challenge. The majority of focus group participants — program and 
control group members alike — said that they had to work while in school, making it difficult 
for them to enroll in courses full time. In the words of one student who had dropped from full 
time to part time, “I have school and home and work conflicting and competing for my time. It’s 
anxiety.” Many students said that they had a number of family responsibilities, such as caring 
for children or supporting their families financially. For study participants in both groups who 
were no longer enrolled, family responsibilities were the most common reason why they had left 
school. In fact, several of them, enrolled and not, said that their families were not supportive of 
their college plans, and instead thought they should enter the workforce after high school to help 
support the family financially; coaches said that this pressure was especially acute for students 
whose families faced housing and food insecurity. Such demands made it difficult for students 
to stay enrolled when they also faced other issues that made them feel they did not belong in 
college, for example struggling in class because they were underprepared academically. Among 
program group members who had dropped out of school, focus group participants who had met 
with coaches often said that their coaches had tried to help them before they left college, but that 
their needs went beyond what the program or college could provide. Interviews with coaches cor- 
roborated this finding, as coaches reported that the most significant reasons students dropped 


19 FAFSA is the form used to apply for Pell Grants and other need-based federal financial aid. 
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TABLE 3. Direct Cost of the Program per Program Group Member per Year 


PROGRAM COMPONENT COST ($) PERCENTAGE OF THE TOTAL 


Administration and staffing 


Program administrators 190 18.5 
Other costs 34 3.3 
Subtotal 224 21.9 
Coaching 569 55.4 
Monthly incentives 234 22.7 
Total direct cost 1,027 100.0 


SOURCE: MDRC calculations based on program expenditure data from the Detroit Regional 
Chamber. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 


Program costs are based on total costs during the first two years of the program. The 
discount rate used for program costs is 3 percent. All costs are shown in constant 2018 dollars. 


out of college were not academic. Most often students found they needed to work, or experienced 
a sudden change in circumstances, such as homelessness, that took precedence over education. 


The Costs of the Program 


Detroit Promise Path has several categories of costs directly related to the operation of the pro- 
gram — the direct costs incurred by administering the program and providing the student sup- 
port. Table 3 breaks down these direct costs: administration and staffing, coaching, and student 
financial incentives. The total direct cost per program group student per year is $1,027. (This 
total includes program group members who did not enroll.) Direct costs comprise the bulk of the 
total cost of the program, which is presented in Appendix B. 


About 22 percent of the direct cost of the program — $224 per program group student per year — 
comes from administration and staffing: the fully dedicated program manager who oversees the 
operation of the program, supervises coaches, and provides quality control. Other professional 
and contractual services (in the table labeled as “other costs”) are also included in this category. 
Coaching activities (interactions with students, data management, meetings and training, and 
program development) make up 55 percent of the program’s direct cost, at $569 per program 
group member per year. The financial incentives of $50 per month per student, contingent on 
program participation, make up 23 percent of the direct costs, averaging $234 per program group 
member per year. 


These per-student cost averages are lower than they would be if all students stayed enrolled for 
the duration of the program. If these same direct costs were measured per enrolled student per 
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year, they would be almost twice as high as the dollar amounts above. For example, when more 
students stay enrolled in school, more coaches need to be hired to maintain desired caseload 
levels. Appendix Table B.1 presents more information and provides additional cost calculations. 
All cost estimates are based on data through the end of June 2018. The final report will include 
an additional year of cost data and will present a cost-effectiveness analysis, comparing the cost 
per graduate in the program and control groups. 


WHAT’S NEXT FOR DETROIT PROMISE PATH 


The encouraging early impacts on enrollment and persistence presented here and in previous 
briefs have led to the expansion of Detroit Promise Path. As of fall 2018, all incoming freshmen 
eligible for the Detroit Promise can join the Detroit Promise Path program at four out of the five 
area colleges, with a goal of making the program universal in the coming years. The one- and 
two-year academic impacts presented in this report indicate that the program has large impacts 
on first-year, full-time enrollment and persistence, while there is still work to be done both on 
initial enrollment and fall-to-fall retention, where there are positive but more modest impacts. 


As part of the Detroit Promise Path’s expansion process, the Chamber has sought out ways to 
continue strengthening the program while serving more students. The Chamber and MDRC used 
a continuous improvement framework to assess and enhance the program’s operation through- 
out the first two years, and the Chamber continues to refine the program over time in response 
to program findings and staff members’ experiences with students. Among the changes already 
implemented are: 


e Amore sophisticated use of the management information system, including more targeted mes- 
sages informed by behavioral science and program reports (for example, identifying groups of 
students who all need to complete their FAFSAs or all need to attend a second monthly coaching 
session, in order to make the messages going out to them more relevant) 


¢ The inclusion of group coaching sessions to build a sense of camaraderie among program 
students and connect them with other students on campus 


¢ More focus on building connections with campus staff members to alleviate financial aid issues 
(so that, for example, student aid issues can be identified and dealt with before the semester 
starts and students are dropped for nonpayment) 


The program is considering ways to use text messaging to help alleviate “summer melt” issues, 
following the example of other postsecondary texting interventions that have shown positive 
results.2° This work began with the goal of helping students with the transition from high school 


20 Summer melt is a term used to describe the high frequency with which students intending to enroll in college in 
the fall semester disengage during the summer and do not enroll after all. Students may fail to enroll because 
they lose motivation, face issues with enrollment or financial aid, or experience a change in circumstances 
leading them to join the workforce instead, among other reasons. For results from texting-based interventions, 
see Castleman et al. (2017); Castleman and Page (2016); and Headlam, Anzelone, and Weiss (2018). 
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to college, but is expanding to summer melt between the first and second years of college, espe- 
cially for students who are not enrolled in summer courses. Helping more students stay enrolled 
through summer engagement could help improve fall-to-fall retention rates and eventually 
graduation rates, too. 


Programs that aim to improve both access and success, as Detroit Promise Path is doing, have a 
unique ability to affect students’ behavior and support them continually throughout their college 
careers. Connecting with students earlier in high school and continuing that outreach through- 
out their years in college can help them get acclimated to college initially, then also navigate 
problems that arise later. This continuing support is especially valuable in an area like Detroit, 
where high levels of poverty can make it hard for students to succeed in college. As noted above, 
in focus groups students and staff members identified a number of external factors contributing 
to students’ difficulty staying in school that go beyond what typical college programs can alle- 
viate — not just academic underpreparedness but also competing responsibilities such as work 
and child care, transportation issues, and food and housing insecurity. Earlier interventions and 
connections to other community programs to help with these issues are among the steps Detroit 
Promise Path is taking to support students staying in school. 


Implications for College Promise Nationwide 


As mentioned above, there are now more than 300 College Promise programs nationwide. These 
programs predominantly focus on college access: giving students the financial resources to attend 
college, and making sure students know these resources exist so they can make the decision to 
enroll. But it is clear from the research literature on low-income students’ experiences in college 
that access alone is not enough. This evaluation of the Detroit Promise Path shows that adding 
student support services to an existing scholarship can have a meaningful impact on students’ 
academic progress. Remember that this evaluation tests the new program compared with a control 
group who continued to receive the Detroit Promise scholarship. For these students, providing 
extra support in the form of campus coaching, outreach, and financial incentives, plus encourag- 
ing enrollment in the summer semester, appears to have improved their educational outcomes. 
Of course, there is still room for improvement when it comes to keeping students in school, but 
this is an important first step for a vulnerable population. 


MDRC’s evaluation of Detroit Promise Path continues through 2021, when the final report showing 
three-year findings for all cohorts will be published. In the interim, College Promise programs 
— and other college programs looking to improve their offerings to students — can learn from 
the work of the Detroit Promise Path. While Promise programs cannot combat every barrier to 
academic success faced by students in poverty, they can contribute to improving college access 
and success for the nation’s low-income students. 
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APPENDIX 


A 


Additional Analyses of Detroit Promise Path 
Impacts on Enrollment and Credits 


his appendix supplements the information provided in the report text with additional ex- 

hibits on enrollment, full-time enrollment, credits attempted, and credits earned. Appendix 

Table A.1 presents a comparison of program and control group outcomes by semester. 

Appendix Table A.1 also presents Detroit Promise Path’s estimated impact among program 
students who ever enrolled in college. Appendix Figure A.1 depicts in a bar chart the estimated 
impacts of Detroit Promise Path among students who ever enrolled in college. Appendix Table 
A.2 presents variation in total credits earned after one year, by subgroups defined by college of 
random assignment, whether a one of a student’s parents attended college, whether a student ap- 
plied for the Detroit Promise scholarship on time (as a proxy for college preparedness), and study 
cohort. Students’ colleges of random assignment and information on their parents’ college statuses 
are drawn from their Detroit Promise scholarship applications. Appendix Table A.3 presents an 
analysis of credits earned in which credit accumulation has been grouped categorically, mak- 
ing it possible to compare the proportions of the program and control groups who successfully 
completed full-time, part-time, or less-than-part-time course loads. 


SUBGROUP ANALYSIS OF STUDENTS WHO ENROLLED AT ANY 
COLLEGE DURING THE STUDY PERIOD 


The primary analyses in this report focus on the intent-to-treat effect: the effect of the option 
to participate in Detroit Promise Path, whether or not program group students actually partici- 
pated in the program. Intent-to-treat analyses are valuable because Detroit Promise Path is not 
mandated, so some students opt not to take advantage of it. The intent-to-treat effect may reflect 
the effect that a policymaker can realistically expect to achieve, given the inevitable scenario that 
not everyone offered a program will participate in it. 


Appendix Table A.1 presents an exploratory analysis estimating the effect of Detroit Promise 
Path among the subset of program group students who received a meaningful “dose” of the 
program’s services (similar to a treatment-on-the-treated analysis). (A meaningful “dose” here 
means primarily that students attended some coaching sessions, but also means that they at- 
tended coaching sessions with some frequency and earned some financial incentives.) To do so, 
the analysis focuses on the effect of Detroit Promise Path among the subset of program group 
students who ever enrolled in college,’ since those who never enrolled in college received little 
or none of the program. 


Stated differently, the table estimates the effect for the type of student who enrolls when offered 
the option to participate in Detroit Promise Path. This analysis assumes that Detroit Promise Path 
had no effect on program group students who did not enroll at any college, a weak assumption.” 


1 This subset includes those who enrolled at a nonstudy college, which can be determined using data from 
the National Student Clearinghouse. 


2 If this assumption is wrong, however, the estimated effects can only be biased downward, so the results 
presented here would be underestimates. 
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Presumably, enrollees were much more likely than average to experience a “reasonable” amount 
of Detroit Promise Path services. These results can be found in the final column of Appendix 
Table A.1 under the header “Effects on Enrolled Students.” 


Among program group students who enrolled at any time during the study period, the estimated 
effect of Detroit Promise Path on enrollment is about 6 percentage points in the first program 
semester and about 10 percentage points in the second semester. The equivalent numbers are 5 
percentage points and 8 percentage points in the intent-to-treat analysis. In other words, among 
program group students who ever enrolled, the positive effects on first-semester and second- 
semester enrollment are slightly larger than the effects among the full sample. 


Similarly, the effect on full-time enrollment is about 8 percentage points in the first program 
semester, and about 13 percentage points in the second program semester. For each of these 
outcomes, the estimated effect among program students who ever enrolled in college is about 26 
percent higher than the estimate of the effect of the option to participate in Detroit Promise Path 
among all program group students. 
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APPENDIX TABLE A.2 Total Credits Earned After One Year: Variation in Impacts by Subgroups 


AVERAGE CREDITS EARNED P-VALUE FOR 
6 P-VALUE DIFFERENTIAL 
SAMPLE PROGRAM CONTROL FOR THE ESTIMATED 
STUDENT CHARACTERISTIC SIZE GROUP GROUP DIFFERENCE DIFFERENCE IMPACTS 
Study college 0.2540 
College 1 573 8.9 7A 1.7" 0.0190 
College 2 149 7.0 5.6 1.5 0.3050 
College 3 185 7.6 4.3 3.2** 0.0110 
College 4 105 7.9 5.0 2.9% 0.0970 
College 5 256 9.5 10.1 -0.7 0.6100 
Sample size 1,268 
Living with a parent who has 0.7740 
earned a bachelor’s degree 
No 987 7.9 6.0 Lo 0.0010 
Yes 247 12.0 9.7 2.3* 0.0840 
Sample size 1,234 
Timeliness of scholarship 0.4970 
application completion 
Before the deadline 829 8.8 6.9 1.9°°* 0.0020 
After the deadline 438 8.1 6.9 1.2 0.1720 
Sample size 1,267 
Study cohort 0.9410 
Fall 2016 589 9.3 77 1.6" 0.0420 
Fall 2017 644 7.9 6.3 1.5** 0.0220 
Sample size 1,233 


SOURCE: MDRC calculations using data from the Detroit Promise Path colleges. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 


A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated 
as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


To test whether the effects vary among subgroups (that is, whether there are differential effects), the HT statistic 
was computed as described by Greenberg, Meyer, and Wiseman (1994) and compared with a chi-squared distribution. 
Statistical significance levels are indicated as: ttt = 1 percent; Tt = 5 percent; t = 10 percent. 


Estimates are adjusted by site, cohort, gender, ACT or SAT score, and the interaction of gender and race/ethnicity. 
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APPENDIX TABLE A.3 Credit Accumulation in the First Two Years of the Study 


PROGRAM CONTROL DIFFERENCE STANDARD 
OUTCOME GROUP GROUP (IMPACT) ERROR P-VALUE 
Total credits earned in Year 1 
0 40.8 50.1 -9.2*** 2.8 0.0010 
1 to 11 23.7 21.2 2.5 2.5 0.3150 
12 to 23 24.6 23.1 1.5 2.5 0.5540 
24 or more 10.8 5.6 5.3*** 1.5 0.0010 
Sample size (total = 1,268) 829 439 
Total credits earned in Years 1 
and 2 
0 34.8 43.4 -8.6** 4.2 0.0410 
1 to 23 38.2 35.2 3.0 4.2 0.4770 
24 to 47 22.1 18.4 3.7 3.3 0.2670 
48 or more 4.9 3.0 1.9 1.7 0.2530 
Sample size (total = 589) 390 199 


SOURCE: MDRC calculations using data from the Detroit Promise Path colleges. 


NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; *“ = 5 percent; * = 10 percent. 

Estimates are adjusted by site, cohort, gender, ACT and/or SAT score, and the interaction of gender and race/ 
ethnicity. 

Weights are calculated to make the effective (weighted) random assignment ratio the same in all random 
assignment blocks. The effective random assignment ratio is equal to the full sample’s random assignment ratio. 
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APPENDIX FIGURE A.1 Detroit Promise Path Increases Enrollment 
Among Students Who Enrolled in College 
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SOURCE: MDRC calculations using data from National Student Clearinghouse and 
transcript data from the Detroit Promise Path colleges. 


NOTES: A two-tailed t-test was applied to differences between research groups. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Estimates are adjusted by site, cohort, gender, ACT and/or SAT score, and the 
interaction of gender and race/ethnicity. 

Weights are calculated to make the effective (weighted) random assignment ratio the 
same in all random assignment blocks. The effective random assignment ratio is equal 
to the full sample’s random assignment ratio. 

Full-time enrollment is defined as enrollment in 12 or more credits at a student’s 
college of random assignment. 
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APPENDIX 


B 


Detroit Promise Path Program Costs 


This appendix supplements the information provided in the main report text with additional cost 
calculations, including base costs for status-quo services, indirect or induced costs, and the net 
cost of the program. In addition, alternate calculations of direct cost are provided. 


COMPUTING DIRECT COSTS 


The direct costs of providing the program’s services cover administration and staffing, student 
services, and financial support. Table 3 in the brief shows the total annual direct cost per sample 
group member for Detroit Promise Path ($1,027). This estimate spreads costs across all students 
who were assigned to the evaluation’s program group, including those who enrolled less than full 
time, dropped out, or graduated. Cost results are described using this approach (rather than using 
a cost per full-time equivalent approach) in order to align cost estimates with the outcomes and 
effects that are described in the impact section of this brief, which also include all sample members. 


Direct costs per student per year were calculated by taking the total cost of the program to date 
and dividing it by the number of students who were offered the program (study sample members 
as well as those not in the analysis sample who were offered the program) and dividing again by 


the average number of academic years since the beginning of the program (approximately two).' 


Direct cost per student per year = (total program cost) / (number of students offered the 
program * average years since entering program) 


DEFINITIONS OF DIRECT COST CATEGORIES 


In Detroit Promise Path, administration and staffing costs consist of the following: 

e Administration — the salaries, benefits, and overhead of the program management staff mem- 
bers and oversight at the Chamber? 

¢ Other — office supplies, travel, marketing materials, computers 

Student services costs in Detroit Promise Path consist of: 

e Coaching — the salaries, benefits, and overhead of the five campus coaches 


Financial support costs in Detroit Promise Path consist of: 


e The monthly incentive — a monthly $50 reloadable gift card, contingent on participation 


1 There are 14 individuals in the Detroit Promise Path study who are not part of the impact analysis sample. 
This number includes people who were not randomly assigned in order to ensure that they were placed 
in the same group as a sibling who had previously joined the study — in other words, the study assigned 
siblings to the same research group. These students are included in the denominator for cost calculations. 


2 Overhead refers to costs that are not direct labor or direct materials costs. 
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COMPUTING NET COSTS 


Appendix Table B.1 describes the base cost and the indirect (or induced) costs of the program, 
and uses those numbers to arrive at a net cost. 


APPENDIX TABLE B.1. Net Cost of Education per Sample Member per Year 


PROGRAM CONTROL DIFFERENCE 


FEATURE ($) GROUP GROUP (NET) 
Direct cost: cost of primary program components 1,027 0 1,027 
Base cost: cost of credits attempted in the absence of the program 4,472 4,472 0 
Indirect cost: cost of additional credits attempted due to the program 483 0 483 
Upper bound: marginal cost equal to average cost? 967 0 967 
Lower bound: marginal cost equal to zero” 0 0 0 
Total cost 5,983 4,472 1,510 
Upper bound: marginal cost equal to average cost* 6,466 4,472 1,994 
Lower bound: marginal cost equal to zero” 5,499 4,472 1,027 


SOURCE: MDRC calculations based on expenditure, transcript, and scholarship data from the Detroit Regional 
Chamber, and financial and enrollment data from the Integrated Postsecondary Education Data System. 


NOTES: Tests of statistical significance were not performed. 

Rounding may cause slight discrepancies in sums and differences. 

Program costs exclude external research costs. 

Credits attempted include all college-level and developmental credits attempted. 

“Marginal cost equal to average cost” represents the case in which existing college resources cannot be 
used to accommodate changes in credits attempted, so the college incurs additional costs for each new credit 
attempted equal to the average cost per credit attempted at the institution. 


>«Marginal cost equal to zero” represents the case where the college can absorb the additional credits 
attempted by the program group without increasing new costs. 


Calculating Base Cost 


The base cost is an estimate of the cost of the “usual” college services provided to students who 
are not in Detroit Promise Path — the cost of instructors, buildings, college administration, etc. 
The base cost provides context for interpreting the programs’ direct cost. 


This analysis uses the estimated cost of credits attempted as a proxy for base costs. This approach 
assumes that resource use corresponds to the number of credits attempted; in other words, a stu- 
dent who attempts more credits is generally associated with greater expenditures than a student 
who attempts fewer credits. “Credits attempted” is a good measure to use to estimate base costs 
because it provides a simple gauge of a student’s level of engagement with the college. To estimate 
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the dollar value of credits attempted, the number of credits attempted is multiplied by an estimated 
cost per credit.* The cost of the usual college experience estimated using this calculation comes 
to about $4,472 per year for each control group student. 


One limitation of this approach is that it assumes that all credits attempted have the same cost to 
the college, which is probably not the case. For example, science lab courses may be more expen- 
sive than English courses. The analysis also assumes that the average cost of serving a student 
at the college is similar to the average cost of serving a student in the study sample. These both 
seem to be reasonable assumptions for this analysis because the process of random assignment 
helps ensure that any differences between the true cost of credits attempted and the average cost 
used in the calculation affect the program and control group similarly. 


Calculating the Indirect Costs of the Program 


Because there is a cost attached to taking more credits, if the program induces students take more 
credits (as is the case in the present study), then the college will incur additional costs. These 
are referred to as indirect costs. While it is likely that a small number of program group students 
taking additional credits would not actually cost the college any more, at some point if enough 
students started to enroll who would not have otherwise, or attempted a full-time load instead 
of a part-time one, the college would need to offer more courses and hire more staff members. 


The indirect cost is estimated based on the average number of additional credits attempted by the 
program students compared with the control group students. This analysis is conducted using 
three different approaches, resulting in a lower bound, an upper bound, and a midpoint between 
the two. The lower-bound estimate assumes that the indirect costs equal zero — that is, that the 
college incurs no additional cost when more students enroll or when students attempt additional 
credits. An upper bound is based on average costs. For example, if students are enrolling in ad- 
ditional courses that are filled to capacity, then the college may have to open new course sections, 
which would cost more. It is unlikely that every additional credit attempted by a student costs the 
college as much as the average credit attempted, and it is also unlikely that there is zero cost to 
the college for additional credits attempted. The midpoint between the upper- and lower-bound 
estimates is therefore used as the primary estimate of indirect costs. 


That midpoint is $483 per student per year. This number is intended to approximate the indirect 
costs should this program continue, as well as to provide a useful estimate to other colleges. 
However, for the time period covered in this report, the colleges in this study were facing under- 
enrollment challenges, so the indirect credit cost may have been closer to the lower bound of $0. 


3 Students in the control group attempted approximately 9.6 credits per year from the time of random 
assignment through June 2018. The cost per credit (approximately $464) is estimated by dividing 
the college’s annual total expenses and deductions by total instructional activity (credit and contact 
hours attempted) at the college during the year of interest. Expense data are taken from the Integrated 
Postsecondary Education Database System of the National Center for Education Statistics. The values 
include the cost of depreciation and cost of scholarships. 
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Moreover, it is also worth noting that, from the colleges’ perspective, indirect costs are offset by 
increased revenue in the form of increased tuition associated with the additional credits attempted. 


Part of the indirect cost is covered by additional Promise scholarships extended to students who 
enrolled because of the program when they would not have otherwise. This cost is calculated 
by comparing the scholarship funds expended for students in the control group with the funds 
expended for those in the program group. The Chamber ended up paying additional scholarship 
costs of about $100 per program group student per year (about a fifth of the total indirect cost). 


Calculating the Total and Net Costs 


The costs of each group are presented in the total line of Appendix Table B.1. The total cost is 
calculated by adding the direct cost, base cost, and indirect costs. The total cost of the program 
per program group member per year was $5,983, compared with the $4,472 cost to educate the 
average control group member. The net cost is defined as the difference between the total pro- 
gram group cost and the total control group cost. The net yearly cost is $1,510 per program group 
member, representing a 34 percent increase. 


DIRECT COST PER ENROLLED STUDENT PER YEAR 


The cost per program group member per year may be of interest to those seeking to create a 
budget for the three-year costs of operating this type of program for a cohort of students (or for 
aligning net costs with the effects of the programs). However, the cost per program group mem- 
ber underestimates the amount spent on enrolled students, since it includes all students (enrolled 
and unenrolled) in the calculation, and many students drop out or graduate within a three-year 
period. Consequently, some readers may be interested in understanding the cost per enrolled 
student per year, since at many colleges a large proportion of revenue is associated with enroll- 
ment. Moreover, a college seeking to sustain a program of this type may want the cost of serving 
a particular number of students per year, with the plan of backfilling slots as students drop out or 
graduate. For these (and other) reasons, the cost per enrolled student per year may be of interest. 


The cost per program member per semester enrolled was calculated using enrollment data for 
program group students. The number of semesters that each student took courses was averaged 
to attain this number (approximately one and a half). This amount serves as a proxy of cost per 
program participant since students who do not enroll are not receiving the program. Using this 
method, the direct cost is $2,101 per enrolled student per year, about $1,000 higher than (or 
double) the direct cost per program group student per year (that is, when students who did not 
enroll were included). This amount may be a more accurate reflection of what the program actu- 
ally spent on participating students.* 


4 Base costs for students who would be enrollees if given the opportunity to be in the program are not 
estimated, nor are net costs for enrollees. The credits attempted by the equivalent control group students 
(who would have enrolled had they been in the program group) are not directly observable because the 
intervention affected which students enrolled, even in the first semester. 
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As mentioned in the main text, the final report will include an additional year of cost data, al- 
lowing the above analyses to be updated, and will present a cost-effectiveness analysis, comparing 
the cost per graduate in the program and control groups. 
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ABOUT MDRC 


MDRC IS A NONPROFIT, NONPARTISAN SOCIAL AND EDU- 
CATION POLICY RESEARCH ORGANIZATION DEDICATED TO 
learning what works to improve the well-being of low-income 
people. Through its research and the active communication of its 
findings, MDRC seeks to enhance the effectiveness of social and 
education policies and programs. 


Founded in 1974 and located in New York; Oakland, California; 
Washington, DC; and Los Angeles, MDRC is best known for 
mounting rigorous, large-scale, real-world tests of new and ex- 
isting policies and programs. Its projects are a mix of demon- 
strations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. 
MDRC’s staff members bring an unusual combination of research 
and organizational experience to their work, providing expertise 
on the latest in qualitative and quantitative methods and on pro- 
gram design, development, implementation, and management. 
MDRC seeks to learn not just whether a program is effective but 
also how and why the program’s effects occur. In addition, it tries 
to place each project’s findings in the broader context of related 
research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, 
and best practices are shared with a broad audience in the policy 
and practitioner community as well as with the general public and 
the media. 


Over the years, MDRC has brought its unique approach to an 
ever-growing range of policy areas and target populations. 
Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, 
employment programs for ex-prisoners, and programs to help 
low-income students succeed in college. MDRC’s projects are 
organized into five areas: 


e Promoting Family Well-Being and Children’s Development 
e Improving Public Education 

e Raising Academic Achievement and Persistence in College 
e Supporting Low-Wage Workers and Communities 

e Overcoming Barriers to Employment 


Working in almost every state, all of the nation’s largest cities, 
and Canada and the United Kingdom, MDRC conducts its proj- 
ects in partnership with national, state, and local governments, 
public school systems, community organizations, and numerous 
private philanthropies. 
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